Skip to content

[LEADS-504] Simplify Ragas context precision metric names#280

Open
bsatapat-jpg wants to merge 1 commit into
lightspeed-core:mainfrom
bsatapat-jpg:met_rename
Open

[LEADS-504] Simplify Ragas context precision metric names#280
bsatapat-jpg wants to merge 1 commit into
lightspeed-core:mainfrom
bsatapat-jpg:met_rename

Conversation

@bsatapat-jpg

@bsatapat-jpg bsatapat-jpg commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Description

Type of change

  • Refactor
  • New feature
  • Bug fix
  • CVE fix
  • Optimization
  • Documentation Update
  • Configuration Update
  • Bump-up service version
  • Bump-up dependent library
  • Bump-up library or tool used for development (does not change the final image)
  • CI configuration change
  • Unit tests improvement

Tools used to create PR

Identify any AI code assistants used in this PR (for transparency and review context)

  • Assisted-by: Opus-4.6-high
  • Generated by: Claude

Related Tickets & Documents

Checklist before requesting a review

  • I have performed a self-review of my code.
  • PR has passed all pre-merge test jobs.
  • If it is a core feature, I have added thorough tests.

Testing

  • Please provide detailed steps to perform tests related to this code change.
  • How were the fix/results from this change verified? Please provide relevant screenshots or results.

Summary by CodeRabbit

  • New Features
    • Introduced context utilization and streamlined context precision metrics as the primary options, updating the default evaluation configuration.
  • Bug Fixes
    • Improved scoring reliability by clamping metric values to the valid 0–1 range to prevent downstream validation failures.
    • Added backward-compatible handling for legacy metric names, with deprecation warnings.
  • Documentation
    • Updated the evaluation guide, README, and example metric configurations to reflect the new/renamed context metrics and requirements.

@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@bsatapat-jpg, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 55 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 55f06d9b-b95d-4bf9-b384-7bb27a36122d

📥 Commits

Reviewing files that changed from the base of the PR and between 2d6ba6b and 367cb2f.

📒 Files selected for processing (9)
  • README.md
  • config/evaluation_data.yaml
  • config/system.yaml
  • docs/EVALUATION_GUIDE.md
  • examples/02_metrics/context_quality/README.md
  • examples/02_metrics/context_quality/system.yaml
  • src/lightspeed_evaluation/core/constants.py
  • src/lightspeed_evaluation/core/metrics/ragas.py
  • src/lightspeed_evaluation/core/system/validator.py

Walkthrough

Updates Ragas context metric naming across implementation, validation, configuration, examples, and documentation. Adds deprecated-name compatibility for old metric identifiers, while consolidating context precision into context_precision and adding context_utilization.

Changes

Context precision metric consolidation

Layer / File(s) Summary
Ragas metric implementation and score clamping
src/lightspeed_evaluation/core/constants.py, src/lightspeed_evaluation/core/metrics/ragas.py
Adds deprecated metric mapping, warning-based alias resolution, clamping, and the consolidated context_precision/context_utilization implementations.
Validator metric requirements and alias resolution
src/lightspeed_evaluation/core/system/validator.py
Canonicalizes deprecated metric names during validation and adds the consolidated ragas:context_precision requirement.
Config, examples, and docs updates
config/system.yaml, config/evaluation_data.yaml, examples/02_metrics/context_quality/system.yaml, examples/02_metrics/context_quality/README.md, README.md, docs/EVALUATION_GUIDE.md
Aligns defaults, metadata, examples, and documentation with ragas:context_precision and ragas:context_utilization.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly reflects the main refactor: simplifying and renaming Ragas context precision metric names.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/lightspeed_evaluation/core/metrics/ragas.py (1)

221-243: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Missing turn_data is None guard, unlike sibling _evaluate_context_precision.

_evaluate_context_precision (added right below in this same diff) and _evaluate_context_recall both short-circuit with a clear error when turn_data is None. _evaluate_context_utilization doesn't, so a None turn_data silently falls through _extract_turn_data to empty query/response/contexts and triggers a real LLM call with empty inputs instead of a clean, immediate error message.

🩹 Suggested fix for consistency
         if is_conversation:
             return None, "Context utilization is a turn-level metric"

+        if turn_data is None:
+            return None, "TurnData is required for context utilization"
+
         query, response, contexts = self._extract_turn_data(turn_data)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lightspeed_evaluation/core/metrics/ragas.py` around lines 221 - 243, Add
the same explicit turn_data None guard to _evaluate_context_utilization that
_evaluate_context_precision and _evaluate_context_recall use, so it fails fast
with the same clear error instead of calling _extract_turn_data and
ContextUtilization.score with empty inputs. Keep the behavior consistent with
the sibling metric helpers in ragas.py by checking turn_data before unpacking
query/response/contexts and returning the same immediate error path.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/lightspeed_evaluation/core/metrics/ragas.py`:
- Around line 221-243: Add the same explicit turn_data None guard to
_evaluate_context_utilization that _evaluate_context_precision and
_evaluate_context_recall use, so it fails fast with the same clear error instead
of calling _extract_turn_data and ContextUtilization.score with empty inputs.
Keep the behavior consistent with the sibling metric helpers in ragas.py by
checking turn_data before unpacking query/response/contexts and returning the
same immediate error path.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ebb2c3bc-950c-499a-ac25-09b5748c8302

📥 Commits

Reviewing files that changed from the base of the PR and between 807df32 and 231fa96.

📒 Files selected for processing (5)
  • config/evaluation_data.yaml
  • config/system.yaml
  • examples/02_metrics/context_quality/system.yaml
  • src/lightspeed_evaluation/core/metrics/ragas.py
  • src/lightspeed_evaluation/core/system/validator.py

@xmican10 xmican10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks! Minor nit, please update the documentation so it is in sync with the updates.

xmican10
xmican10 previously approved these changes Jul 7, 2026

@asamal4 asamal4 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not just a rename change - we need to have backward compatibility with deprecation notice in the code..
Old long names continue to work as aliases (log a deprecation warning)

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/lightspeed_evaluation/core/system/validator.py (1)

468-472: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate canonicalization logic.

The canonical = DEPRECATED_METRIC_NAMES.get(metric, metric) + dual-membership check is repeated verbatim for turn-level and conversation-level validation. Consider extracting a small helper, e.g. _is_metric_known(metric, known_metrics), to avoid drift if the logic changes later.

♻️ Proposed refactor
+    `@staticmethod`
+    def _is_metric_known(metric: str, known_metrics: set[str]) -> bool:
+        """Check if a metric (or its canonical alias) is in the known set."""
+        canonical = DEPRECATED_METRIC_NAMES.get(metric, metric)
+        return metric in known_metrics or canonical in known_metrics
+
     def _validate_metrics_availability(self, data: EvaluationData) -> None:
         ...
         for turn_data in data.turns:
             if turn_data.turn_metrics:
                 for metric in turn_data.turn_metrics:
-                    canonical = DEPRECATED_METRIC_NAMES.get(metric, metric)
-                    if (
-                        metric not in self._turn_level_metrics
-                        and canonical not in self._turn_level_metrics
-                    ):
+                    if not self._is_metric_known(metric, self._turn_level_metrics):
                         turn_data.add_invalid_metric(metric)
                         ...

Also applies to: 482-486

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lightspeed_evaluation/core/system/validator.py` around lines 468 - 472,
The turn-level and conversation-level metric checks in validator.py duplicate
the same canonicalization and membership logic. Extract that repeated logic into
a small helper such as _is_metric_known(metric, known_metrics) in Validator, and
have both validation branches call it instead of inlining
DEPRECATED_METRIC_NAMES.get(...) and the dual-membership check. Keep the helper
generic so both _turn_level_metrics and the conversation-level known metrics use
the same source of truth and future changes stay consistent.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lightspeed_evaluation/core/system/validator.py`:
- Around line 468-472: The turn-level and conversation-level metric checks in
validator.py duplicate the same canonicalization and membership logic. Extract
that repeated logic into a small helper such as _is_metric_known(metric,
known_metrics) in Validator, and have both validation branches call it instead
of inlining DEPRECATED_METRIC_NAMES.get(...) and the dual-membership check. Keep
the helper generic so both _turn_level_metrics and the conversation-level known
metrics use the same source of truth and future changes stay consistent.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 1aa56e71-37ca-480e-aa50-d7cbba2cff0a

📥 Commits

Reviewing files that changed from the base of the PR and between 231fa96 and 2d6ba6b.

📒 Files selected for processing (9)
  • README.md
  • config/evaluation_data.yaml
  • config/system.yaml
  • docs/EVALUATION_GUIDE.md
  • examples/02_metrics/context_quality/README.md
  • examples/02_metrics/context_quality/system.yaml
  • src/lightspeed_evaluation/core/constants.py
  • src/lightspeed_evaluation/core/metrics/ragas.py
  • src/lightspeed_evaluation/core/system/validator.py
✅ Files skipped from review due to trivial changes (2)
  • README.md
  • examples/02_metrics/context_quality/system.yaml
🚧 Files skipped from review as they are similar to previous changes (3)
  • config/evaluation_data.yaml
  • src/lightspeed_evaluation/core/metrics/ragas.py
  • config/system.yaml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants